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Abstract. We study quadratic optimal stochastic control problems with control dependent noise 
state equation perturbed by an affine term and with stochastic coefficients. Both infinite horizon 
case and ergodic case are treated. To this purpose we introduce a Backward Stochastic Riccati 
Equation and a dual backward stochastic equation, both considered in the whole time line. Besides 
some stabilizability conditions we prove existence of a solution for the two previous equations 
defined as limit of suitable finite horizon approximating problems. This allows to perform the 
synthesis of the optimal control. 
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1. Introduction 

Backward Stochastic Riccati Equations (BSREs) are naturally linked with stochastic optimal 
control problems with stochastic coefficients. The first existence and uniqueness result for such a 
kind of equations has been given by Bismut in [3], but then several works, see [I], [14| . [15] . [IS] . 
[IT] . [19] and [20], followed as the problem, in its general formulation, turned out to be difficult to 
handle and challenging. Indeed only very recently Tang in [22\ solved the general non singular case 
corresponding to the linear quadratic problem with random coefficients and control dependent noise. 

In his paper the so-called linear quadratic optimal control problem is considered: minimize over 
all admissible controls u the following cost functional 



J{0,x,u) = E 



/ [(S s X s ,X s ) + \u s \ 2 }ds + E(PX T ,X T )- (1.1) 
Jo 

and inf M J(0, x, u) is the so called value function. In (jl.lj) . X s 6 R" is solution of the following linear 
stochastic system: 

( d 

dX s = (A s X s +B s u s )ds + J2( C i x s+Diu s )dWi s>0 (J 2) 

i=l 

X = X, 

where W is a d dimensional brownian motion and A, B, C, D, S are stochastic processes predictable 
with respect to the natural augmented filtration {Tt\t>o while P is a random variable J^-measurable. 
All these results cover the finite horizon case. 

In this paper starting from the results of |22j , we address the infinite horizon case and the ergodic 
case. The infinite horizon case, with random coefficients, has been addressed also in [T2]. In [H] the 
infinite dimensional case is treated but no control dependent noise appears in the state equation. 
Since our final goal is to address ergodic control, in the state equation we consider a forcing term. 
Namely, the state equation that describe the system under control is the following affine stochastic 
equation: 

d 

dX s = (A S X S + B s u s )ds + J2 (C\X S + D\u s ) dWl + f s ds s > ^ g) 

2=1 



Our main goal is to minimize with respect to u the infinite horizon cost functional, 

f + OO 



/•-too 

J oo (0,x,u) = E [{S s X s ,X s ) + \u s \ 2 ]ds 
Jo 



and the following ergodic cost functional: 



liminf aJ a {0,x,u) 

ot— *0 



where 



J a (0,x,u) = E 



+00 



' [(S S X S , X s 



2 }ds 



(1.4) 
(1.5) 
(1.6) 



In order to carry on this programme we have first to reconsider the finite horizon case since now 
the state equation is affine. As it is well known the value function has in the present situation a 
quadratic term represented in term of the solution of the Backward Stochastic Riccati Equation 
(BSRE) in [0,T]: 



dP t 



A* t P t + P t A t + S t + J2(( c , 



i=l 



P t Cl + (Cl)*Q t + Q t C, 
-J2(Dl)*P t Dl 



-1 r 



PtB t 



dt + Y^Q\dW t 

i=l 



PtD\ 



Q\D\ 



1.7) 

* 

dt, 



Pt = P, 



and a linear term involving the so-called costate equation (dual equation) : 

d d 

dv T t = -Hjrfdt - Ptftdt - £ {KlYgfdt + Yj\ T dWi, t G [0, T] 

i=i i=i 

r% = 0. 



fl.S 



The coefficients H and K are related with the coefficients of the state equation and the solution to 
the BSRE on [0,T]. In details, if we denote for t <E [0,T] 

d d 

a (*, p u Q t ) = -[i + Y J (diY PtDir^PA + j2(QiDi + (c*y p t D\)),* 



1=1 



i=l 



then we have: H t = A t + B t A {t, P u Q t ) and K\ = C\ + D\K (t, P u Q t ). The solution (r, g) of this 
equation together with the solution (P, Q) of the BSRE equation (|1.7p allow to describe the optimal 
control and perform the synthesis of the optimal equation. Equation (|1.8p is the generalization of 
the deterministic equation considered by Bensoussan in [1] and by Da Prato and Ichikawa in [9] and 
of the stochastic backward equation introduced in 24J for the case without control dependent noise 
and with deterministic coefficients. 

The main difference from the equation considered in is that, being the solution to the Riccati 
equation a couple of stochastic processes (P, Q) with Q just square integrable, equation ()1.8jl has 
stochastic coefficients that are not uniformly bounded. So the usual technique of resolution does not 
apply directly. When r is one dimensional similar difficulties are treated e.g. in [5] using Girsanov 
Theorem and properties of BMO martingales. Here being the problem naturally multidimensional 
we can not apply the Girsanov transformation to get rid of the term (J^t)* dl^t + Yli=i9tdWt . 

Nevertheless we can exploit a duality relation between the dual equation (|1.8[) and the following 
equation 

d 

dX s = H s X s ds + ^KlXsdWl s e [t, T) 



(1.9) 



X t = x. 



This equation is indeed the closed loop equation related to the linear quadratic problem and can be 
solved following |llj and its control interpretation allows to gain enough regularity to perform the 
duality relation with (r,g). 
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Once we are able to handle the finite horizon case, we can proceed to study the infinite horizon 
problem. The BSRE corresponding to this problem is, for t > 0, 



dPt = 



A$P t + P t A t + S t + J2 UptT p A + (Q)* Q t + Q t Q 



P t B t + J2((C$)* PtDi + QWt 



i=l 



dt + j2Qidwi+ 



i+j2(D l t yp t Di 



(1.10) 



P t B t + J2((C l t )*PtDl + QlDl 



dt. 



Note that differently from equation fll.7j) . the final condition has disappeared since the horizon is 
infinite. We assume a suitable stabilizability condition, see also [12] . namely we ask that there exists 
a control u <E £^([0, +oo) x fi; R k ) such that for all i > and all x € R n 



+ 00 



[(S s Xi' x '»,Xl' x ' u ) + M 2 Rs < M t . 



for some positive constant M t>x . By L^QO, +oo) x f2; R fc ) we denote the space of predictable square 
integrable processes. Under this stabilizability condition, we prove that there exists a minimal 
solution (P, Q) and we can perform the synthesis of the optimal control with / = 0. More precisely 
we introduce a sequence (P N , Q N ) of solutions of the Riccati equation in [0, TV] with P N (N) = 
and we show that for any t > the sequence of P N pointwise converge, as N tends to +oo, to a limit 
denoted by P. The sequence of Q N instead only converge weakly in Lp(f2 x [0,T]) to some process 
Q and this is not enough to pass to the limit in the fundamental relation and then to conclude that 
the limit (P,Q) is the solution for the infinite horizon Riccati equation (|1.10[) . Therefore, as for 
the finite horizon case, we have to introduce the stochastic Hamiltonian system to prove that the 
limit (P,Q) solves the BRSE ([l.lOp . see Corollary 13. 71 Indeed studying the stochastic Hamiltonian 
system we can prove that the optimal cost for the approximating problem converge to the optimal 
cost of the limit problem and this implies that P is the solution of the BSRE. 

In order to cope with the affine term / we have to introduce an infinite horizon, this time, 
backward equation 



dr t = 



-H* t r t dt - P t f t dt - 



' g\dt + Y,9ldWl, 



t > 0. 



(1.11) 



i=l 



Notice that the typical monotonicity assumptions on the coefficients of this infinite horizon BSDE 
are replaced by the finite cost condition and the Theorem of Datko. As a consequence of this new 
hypothesis we have that the solution to the closed loop equation considered in the whole positive 
time line with the coefficients evaluated in P and Q, is exponentially stable. 

Hence a solution (f, g) to this equation is obtained as limit of the sequence (r T ,g T ) defined in 



|), indeed using duality and the exponential stability property of the solution to (|1.9p . we can 
prove that the sequence of r T and its limit f are uniformly bounded. Hence, having both (P, Q) 
and {r,g), we can express the optimal control and the value function. 

Eventually we come up with the ergodic case: first of all we set Xf := e~ as X s and := e~ as u s 
and we notice that the functional J a (0, x, u) can be written as an infinite horizon functional in terms 
of X a and /: 

f + 00 



J a (0,x,u)=K / [(S a X?,X?) + K\ 2 ]ds 
Jo 



This allows us to adapt the previous results on the infinite horizon when a > is fixed. 

Then, in order to study the limit (|1.5|l . we need to investigate the behaviour of X a , of the solution 

P a of the Riccati equation corresponding to J (x) := inf„ J a (0, x, u) and the solutions (r a ,g a ) of 



the dual equations corresponding to H a , K a and /" 



ft- In the general case it turns out that 
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the ergodic limit has the following form: 



]m a ^ooiJ a (x) = lim ct ^ 2aE / (rf,f?)ds 

Jo 

r+oa d d 

- K^ a ^ aE / |(I + £ (DlY P«D\y\B:r« + £ <^)| 2 d S . 
- 70 i=i i=i 

When the coefficients of the state equation are deterministic similar problems have already been 
treated: we cite [2], [24] and bibliography therein. In [2] in the state equation all the coefficients 
are deterministic and no control dependent noise is studied, while in [24] only the forcing term / is 
allowed to be random. 

Finally we describe the content of each section: in section 2, after recalling some results of [2*2*] . 
we solve the finite horizon case when the state equation is affine: the key point is the solution of 
the dual equation (|1.8j) . which is studied in paragraph 2.2; in section 3 we solve the infinite horizon 
case with / = 0, in section 4 we study the infinite horizon equation (jl.lljl . in section 5 we complete 
the general infinite horizon case, finally in section 6 we study the ergodic case. 

Acknowledgments. The authors wish to thank Philippe Briand for the very useful discussions 
on the role of BMO martingales in the theory of Backward Stochastic Differential Equations and 
Gianmario Tessitore for very useful discussions. 



2. Linear Quadratic optimal control in the finite horizon case 

Let (ft, £ , (.Ft)t>o> P) be a stochastic base verifying the usual conditions, and let (Tt)t>o be natural 
augmented filtration generated by the Brownian motion. In (ft,£,P) we consider the following 
stochastic differential equation: 

d 

dX s = (A S X S + B s u s )ds + ^ (C\X S + D l s u s ) dWj + f s ds s e [t, T] ^ ^ 

l — l 

X t = x 

where X is a process with values in R™ and represents the state of the system and is our unknown, 
u is a process with values in R k and represents the control, {W t := (W/, ...,Wf) , t > 0} is a d- 
dimcnsional standard Brownian motion and the initial data x belongs to W 1 . To stress dependence 
of the state X on u, t and x we will denote the solution of equation (12. ip by X t,x,u when needed. 
The norm and the scalar product in any finite dimensional Euclidean space R m , m > 1, will be 
denoted respectively by |-| and (•, •). 

Our purpose is to minimize with respect to u the cost functional, 

r T 

J{0,x,u)=E [(S s X°< x ' u ,X°' x < u ) + \u s \ 2 ]ds + E\X% x ' u \ 2 . (2.2) 
Jo 

We also introduce the following random variables, for t G [0, T]: 

J(t,x,u) :=E^ J [(S S X°' X < U ,X°< X ' U ) + \u s \ 2 ]ds + E^ \ X^ x ' u | 2 . (2.3) 

We make the following assumptions on A, B, C and D. 
Hypothesis 2.1. 

Al) A : [0,T] x ft -> W ixn , B : [0,T] x Q —> R nxk , C l : [0,T] X ft -► R nxn , i = l,...,d and 
D l : [0, T] x ft — > M™ xfe , i = 1, d, are uniformly bounded processes predictable. 

A2) S : [0,T] x ft — > R." x ™ is uniformly bounded and predictable and it is almost surely and 
almost everywhere symmetric and nonnegative. 

A3) / : [0, T] x ft — > R n is predictable and f S L°° ([0, T] x ft). VFe denote such space of processes 
L^([0,T]xft). 
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2.1. Preliminary results on the unforced case. Next we recall some results obtained in [55J for 
the finite horizon case, with / = in equation (|2.1|) . In that paper a finite horizon control problem 
was studied, namely minimize the quadratic cost functional 

J(0,x,u) = E(PX°' X ' U ,X°' X ' U ) +E / [(S S X°> X > U ,X°' X > U ) + \u a \ 2 ]ds, 

Jo 

where P is a random matrix uniformly bounded and almost surely positive and symmetric, T > 
is fixed and X°' x,u is the solution to equation (|2.1j) with / = 0. To this controlled problem, the 
following (finite horizon) backward stochastic Riccati differential equation (BSRDE in the following) 
is related: 



-dP t = G (A U B U Ct,D t ; S t -P u Q t ) dt + J^QidWi 

i=l 

P T = P. 



(2.4) 



where 



G {A, B, C, D; S; P, Q) = A*P + PA + S + ({C 1 )* PC 1 + {C 1 )* Q + QC*) - G t (B, C, D; P, Q) 

i=l 

and 



Gt {B, C, D; P, Q) 



PB + Y^ [(C 1 )* PD l + Q l D 
i=i 

d 

pb + Y ((c*)* PL)i + Q iD ' 



I + J2(D 1 ) * PD l 



i=l 



Definition 2.2. A pair of predictable processes (P,Q) is a solution of equation 
f T 2 

(1) / \Q S \ ds < +oo, almost surely 
Jo 

(2) 

/ \G {A S1 B S , C S ,D S ; S s ;P Sl Q s )\ds < +c^, 
Jo 

(3) for all t G [0, T] 

/T t-T d 

G (A s , B S ,C S ,D S ; S s ; P S ,Q S ) ds - J ^jQ^dW*.. 



if 



Theorem 2.3 ([55], Theorems 3.2 and 5.3). Assume that A,B,C,D and S verify hypothesis \2.1\ 
Then there exists a unique solution to equation {2.$ - Moreover the following fundamental relation 
holds true, for allO<t<s< T, and all u € L 2 V ([0, T] x fi, E fc ) : 



(P t x, x) = (PX^ X ' U , X^ x ' u ) + j [(SrX^, Xl> x ' u ) 
I + J2(Dl)*P s Dl 



2 }dr 



(2.5) 



1/2 



f d \ 1 / d \ * 

/ + ]T (Diy p s d\ p s b s + (q'J>:. + «■'.)' rjr) x*™ 

\ i=l / \ i=l / 



ds 



Then the value function is given by 



(P x, x) 



uei4([0 



inf E r ' (PXy, x ' u ,X^ x ' u )+E^ / [(SrXp^Xt^ + luJ 

OJlxfi,!') N ' J t LN ' 



]dr 
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and the unique optimal control has the following closed form: 

-l 



Ui 



I + Y (DlY PtDi] (p t B t + Y (Q\D\ + (CtY PtDl)) X?" 

\ 1=1 / V »=i / 



If X is the solution of the state equation corresponding to u ( that is the optimal state ), then X is 
the unique solution to the closed loop equation: 

/ d d \ 

dX s = A S X S - B S {I + Y (Di)* PtDi)- 1 (PtB t + Y (®t D i + {Cl)*P t Dl))*X s ds+ 

\ i=l i=l ) 

d ( d d \ 

Y cixi - d\{i + (Dty p^i)- 1 (p t B t + Y(Qi D i + ( c iY p tD\)fx s ) dwi 

i=l \ i-1 i=l J 
X t =X 

(2.6) 

The optimal cost is therefore given in term of the solution of the Riccati matrix 

J(0,x,u) = {P x,x). (2.7) 

and also the following identity holds, for all t G [0, T]: 

J(t,x,u) = (P t x,x). (2.8) 

For t & [0, T], we denote by 

A(t,Pt,Qt) = -(i+Y( D t)* p t D i) (ptBt+Y(Q\ D \ + ( c iY p tD\)^ , 

H t =A t + B t A(t,Pt,Qt), 
Kl = Cl + D l t A(t,P t ,Q t ). 

(2.9) 

So the closed loop equation (|2.6|) can be rewritten as 

d 

dx s = H s x s ds + Y K l x sdW> se[t,T) (21Q) 

i—1 

X t = x 

It is well known, see e.g. [TT], that equation (|2.10p admits a solution. 

Remark 2.4. A, H and K defined in (|2.9|) are related to the feedback operator in the solution of the 
finite horizon optimal control problem with / = 0. By the boundedness of P and thanks to standard 
estimates on the Riccati equation, see |22j . theorem 5.3, it turns out that for every stopping time 
< t < T a.s., 

rT 



J \A(t,P u Q t )\ 2 dt<C, 



where C is a constant depending on T and x. Since A, B 7 C and D are bounded, this property 
holds true also for H and K: 



J \H t \ 2 dt + E^ J \K t \ 2 dt<C, 



where now C is a constant depending on T, x, A, B, C and D. In particular, A, H and K are square 
integrable. In the following we denote by 

C H = supE^ [ \H t \ 2 dt, C K = supE^ [ \K t \ 2 dt. (2.11) 

T J T T J T 

where the supremum is taken over all stopping times r, r G [0,T] a.s.. 
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2.2. Costate equation and finite horizon affine control. In order to solve the optimal control 
problem related to the nonlinear controlled equation 12.11 we introduce the so called dual equation, 
or costate equation, 

d d 

dr t = -Hindi - P t f t dt - ( K t) * 9\dt + Yj\dW*, t E [0, T] ^ 

i—1 i—1 

r T = 0. 

At a first step we look for a pair of predictable processes (r,g) satisfying equation (|2.12|) . and s.t. 
reL% c (Q,C([0,T},R n )) and g* € Lf oc ([0,T] x 0,R«), for i = i,...,d. L% c (fi, C([0, T],M«)) is the 
space of predictable processes r with values in R™ that admit a continuous version and such that 

P ( sup \r t \ < oo ] = 1. 
\te[o,T] / 

L 2 oc ([0, T] x J7,R n ) is the space of predictable processes g with values in R™ such that 



^ |g s | 2 ds < ccj = 1. 



Lemma 2.5. TTie backward equation (|2.12[) admits a unique solution {r,g) that belongs to the space 
L% c (n,C({0,T},R n ))xLl c (lO,T}xn,R n * d ). 

Proof. In order to construct a solution to equation (|2.12p . we essentially follow [25], chapter 7, 
where linear BSDEs with bounded coefficients are solved directly. Unlike in [25J, the coefficients of 
equation (|2.12|) are not bounded. Besides equation (|2.12p we consider the two following equations 
with values in R raxn : 

d d 

d$ s = -H s $ s d S +^(Kiy (Kiy$ s ds-^(Kiy$ s dwi; sgm (213) 



i=l 

$0=1, 



and 



d^ s = ^ s H*ds + J2^ s (KlYdW^ se[0,T] (2M) 

i—1 

By applying Ito formula it turns out that $>t*&t = !■ By transposing equation (|2.14|) . we obtain the 
following equation for 'J'*: 

f d 

d** s =H s ** s ds + J2 K t*tdWi se[0,T] (215) 



**=I. 



By [H], equations (|2.13j) . (12.14|l and (|2.15p admit a unique solution. Moreover, since H and K are 
related to the feedback operator, see ()2.9|) where A, H and K are defined, it follows that 

E|*t| 2 < C\I\ 2 ,t £ [0,T], (2.16) 

where C is a constant that may depend on T, see also theorem 2.2 in [22 , with — fa t h, h 6 R n . 

f T 

We set 9 := — I ^ s P s f s ds. By boundedness of P and /, and by estimate (|2.16[) on 'J, it turns 
Jo 

out that 9 e L 2 (Q). We define 



Jo 



Then r e L£ c (fi, C([0,T],R n )), and following [25], chapter 7, theorem 2.2, we can build a pro- 
cess g € L 2 oc (f2,C([0,T],R n )) such that (r,g) is a solution to equation (|2.12|) . with (r,s) € 
LS s (fi,C([0,T],R»)) x L 2 oc ([0,T] x!l,I» xi ). 

□ 
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We can prove that the solution (r, g) to equation (|2.12j) is more regular. To prove this regularity, 
we need the following duality relation. 

Remark 2.6. Let £ € L^r r (0) and let (r, g) be solution to the equation 

d d 

dr t = -Hindi - P t f t dt - ]T (A7) * 9\dt + Y,9\dWi, t e [0, T] 

i=l i=l 

and let X l ' x,ri be solution to the equation 



dXl^'V = H s X^ x ^ds + K\Xl' x *dWt + rj s ds, s € [t, T] , 



(2.17) 



X 



t,x,rj 



where x e L 2 (Vt,T t ) and r/ e £|> (fi x [0,T],R n ). Then, by applying the Ito formula, the following 
duality relation holds true: 



-E^ 



(PJ^X^ds+E^ / ( Vs ,r s )ds 



(2.18) 



We also need to find a relation between the solution (r, g) of the equation (|2.12[) and the opti- 
mal state X corresponding to the optimal control u. This can be achieved, following e.g. [I], by 
introducing the so called stochastic Hamiltonian system 



dX s = [A S X S + B s u s }ds + Y^[CtX s + D\u s ]dWl + f s ds, 



dy s 



[A* s y s + (ClY 4 + S s X s ]ds + J^ztdWj, 



t<s<T, 



(2.19) 



Xf — x, 
, Ut = PtX t , 



i=l 



where y, z 1 € R™, for every i = 1, ...,d. By the so called stochastic maximum principle, the optimal 
control for the finite horizon control problem is given by 



(2.20) 



By relation (j2.20[) . equations (j2 . 19[) become a fully coupled system of forward backward sto- 
chastic differential equations (FBSDE in the following), which admits a unique solution (X,y, z) G 
L%(Sl x [0,T],R n ) x L 2 v {Vlx. [0,T],M") x L% ([0,T] x rj,R" xd ), see Theorem 2.6 in [21]. 

Lemma 2.7. Let (r,g) be the unique solution to equation h2.12\) , and let (X,y,z) be the unique 
solution to the FBSDE \2.19\) . Then the following relation holds true for [0, T]: 



y t = PtXt+n, 



t< s<T. 



(2.21) 



Proof. We only give a sketch of the proof. For t = T relation (|2.2ip holds true. By applying ltd 
formula it turns out that yt — PtX t and r t solve the same BSDE, with the same final datum equal 
to at the final time T. By uniqueness of the solution of this BSDE, the lemma is proved. □ 

We can now prove the following regularity result on (r, g) . 

Proposition 2.8. Let (r, g) be the solution to equation [2J2]) . Then (r, g) e L 2 V (Q, C ([0, T] , W 1 )) x 
L 2 V ([0,T] x n,R nxd ). Moreover r G L^(n x [0,T]). 

Proof. Let (r, g) be the solution to equation (|2.12j) built in lemma 1^31 We note that by theorem 
2.6 in [21], y G Lj, (Q x [0,T],IR n ). Moreover, by standard calculations, it is easy to check that y 
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admits a continuous version and y £ L%, (O, C([0, T], M")). Moreover, if / = 0, we get, for every 
< t < s < T, 

E^LT S | 2 < C\x\ 2 . 

where X is solution to (|2.19[) . This estimate can be easily achieved by applying the Gronwall lemma, 
and by remembering that from (|2.8|) . for the optimal control u the following holds: 

r T 



E^y \u s \ z <{P t x,x) <C\x\ 2 . 

As a consequence, if / 7^ 0, for every < t < s < T, 

E^\X S \ 2 <C(l + \x\ 2 ). 
Since P is bounded, by lemma |2~T1 we get that for every < t < s < T 



sup \r s \ 2 <C, 

t<s<T 



(2.22) 



where C is a constant that can depend on T . In particular, for every < t < T 

\r t \ 2 < C. (2.23) 

Moreover, since X is continuous an P admits a continuous version, also r admits a continuous 
version. By applying Ito formula we get for < t < w < T, 

(H* s r s ,r s ) ds + 2j^ (P s f s ,r s ) ds + 2 jf £ ((^)* «/*,r.) 

+2/ Y,((9iy dw s^s)- J2\9i\ 2ds - ( 2 - 24 ) 

• 7t i=i • 7t »=i 

By estimate (|2.23p and by taking t — 0, we get by standard calculations 



□ 



where C is a constant that may depend on T. 

We are ready to prove the main result of this section 

Theorem 2.9. Assume A, B, C, D and f satisfy hvvothesis \2.1\ Fix x £ W 1 , then: 

(1) there exists a unique optimal control u £ Lj, (fl x [0, T] , K^) such that for every < t < T, 

J(0,x,u)= inf J(0,x,u) 

u6L^(nx[0,T],R fc ) 

(2) If X is the mild solution of the state equation corresponding to u (that is the optimal state), 
then X is the unique mild solution to the closed loop equation: 



dX t 

d 

E 

i=l 
X = X 



t) 9t) 



A t X t - B t A(i, P t ,Qt)X t + + ( D iY PtD\)-\B* t r t + J2( D 
ClX t -Di I A(t,P t ,Q t )X t + (l + ^2(DiyP t Di) (B*r t +Y,(DlY gl) 



i=l 



(3) The following feedback law holds P-a.s. for almost every < t < T . 



dt + f t dt 
dWf, 

(2.25) 



lit = 



I + £ (Di)' P t D\ ) ( P t B t + £ (Q\D\ + (CtyPM) ) Xt + B* t r t + £ (£>j)' 



St- 



(2.26) 
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(4) The optimal cost is given by 

J(0,x,u) = (P x,x)+2(r ,x)-E(P T X T ,X T )+2E f (r s ,f s )ds 

Jo 

-E f '|(J + £(£>')* P t Diy\ B ;r t +J2(Diy g\)\ 2 ds. 
J ° i=l <=i 

Proof. By computing d{PtX t , X t ) + 2(r t , X t ), we get the so called fundamental relation 



f T 

E^ y [(S S A^<", A^* 11 ) + \u s \ 2 ]ds 

= (P t x, x) + 2{r u x) - (P T X T , X t ) + 2R F * J (r s ,f s )ds 

+ ^[(l + E (DIYPsD^ \P S B S + J2 (QlDl + (Cl)*P s Dlfj X s + B* s r s + £ (£>*) ?ds 

E^ [ T \(I + it (DlY P s Dl)-\B* s r s + E (DlYgl)\ 2 ds. 
Jt i=i i=i 

The theorem now easily follows. □ 

3. Preliminary results for the infinite horizon case 

The next step is to study the optimal control problem in the infinite horizon case and with / ^ 0. 
To this aim we have to study solvability and regularity of the solution of a BSRDE with infinite 
horizon, in particular we study P. At first we consider the case when / = 0. Namely, in this section 
we consider the following stochastic differential equation where X l > x > u represents the state: 

d 

dX t,x,u = ( AaX tw + B s u s )ds + V (CiX*™ + D\u s ) dWl s>t fQ-\\ 
, X\' x ' u = x 

As a by product of the preliminaries studies, we are able to solve the following stochastic optimal 
control problem: minimize with respect to every admissible control u the cost functional, 

r+oo 

J oo (0,x,u)=E {(S s X^,X^) + \u s \ 2 ]ds. (3.2) 
Jo 

We define the set of admissible control 

14= j« € L 2 v (n x [0,+oo),R fe ) : E^ + (S S X°> X ' U , X°> x '") + \u s \ 2 ds < +oo| . (3.3) 
We also introduce the following random variables, for t e [0, +oo]: 

/ + oo 
{(SsXl^^Xl-^ + lusftds 

We will work under the following general assumptions on A, B, C and D that will hold from now 
on: 

Hypothesis 3.1. 

Al) A : [0, +oo) x fl -» R nxn , B : [0, +oo) x fl R nxk , C l : [0, +oo) x fl R nxn , i = 1, ...,d 
and D % : [0, +oo) x fl — > R nxk , i = l,...,d, are uniformly bounded process adapted to the 
filtration {Ft} t>0 - 

A2) S : [0, +oo) xfl-t R nxn is uniformly bounded and adapted to the filtration {Ft} t >o an d it 
is almost surely and almost everywhere symmetric and nonnegative. 
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In order to study this control problem in infinite horizon, we consider the following backward 
stochastic Riccati equation on [0, +00): 



dP t 



a 

A* t P t + P t A t + S t + ((Ci)*P t Ci + (Cl)*Q t + Q t C\ 



dt + j2Qi dW i- 



i=l 



a 

p t B t +Y,(( c iY p t Dl t + Q lD i 



-1 -V 



i+j2(Diyp t Di 



i=l 



P t B t + J2((Ci)*P t Di + QiD] 



(3.4) 



dt. 



where we stress that the final condition has disappeared but we ask that the solution can be extended 
to the whole positive real half-axis. 



Definition 3.2. We say that a pair of processes (P, Q) is a solution to equation (|3.4p if for every 
T > (P,Q) is a solution to equation (|2.4p in the interval time [0, T], with with final datum the 
process P evalueted at time T. In particular a solution (P, Q) is called minimal if whenever another 
couple (P, Q) is a solution to the Riccati equation then P — P is a non-negative matrix, see also 
Corollary 3.3 in [H]. 

Definition 3.3. We say that (A, B, C, D) is stabilizable relatively to the observations \/~S (or y/S- 
stabilizable) if there exists a control u S Lj,([0, +00) x ft; U) such that for alH > and all x £ K™ 



[(SiXt^X**'") + \u s \ 2 ]ds < M M . 



(3.5) 



for some positive constant M t , x - 



This kind of stabilizability condition has been introduced in [12] . 

In the following, we consider BSRDEs on the time interval [0, iV], with final condition Pjv = 0. 
For each integer N > 0, let [P N , Q N ) be the solution of the Riccati equation 



~dP t N = G (A t ,B u Ct,D t \ S t ; P t N , Q? ) dt + Yfi^dW} 



(3.6) 



p N — n 
jv — u - 



P can be defined in the whole [0, +00) setting P t = for all t > N . We prove the following 
lemma. 

Lemma 3.4. Assume hypothesis \3.1\ and that (A, B, C, D) is stabilizable relatively to the obser- 
vations V~S. There exists a random matrix P almost surely positive and symmetric such that 
V {\im N ^ +OD P N (t)x = P(t)x, Vx £ W 1 } = 1. 

Proof. The proof essentially follows the first part of the proof of proposition 3.2 in [T^]. □ 



Remark 3.5. It is clear from the above proof of proposition 3.2 in 12 that condition (J375J) is equi- 
valent to the following one: 



[(S s X^ x ' u ,Xl^ u ) + \u s \ 2 ]ds < M\x\< 



(3.7) 



where the constant M may depend on t. 

Next we want to prove that P built in the previous lemma is the solution to the BSRDE 
This is achieved through the control meaning of the solution of the Riccati equation. Indeed also 
the martigale term Q N appears in the fundamental relation is no more possible to proceed as in 
[T2]. For T > fixed and for each N > T, we consider the following finite horizon stochastic optimal 
control problem: minimize the cost, over all admissible controls, 

cT 



J(0, x, u) 



E(P^X^ X - U ,X°' X ' U ) 



E 



[{s s x. 



'*> u ,X° s > x > u ) + \u s \ 2 }ds, 
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where X°- X ' u is solution to equation (|3.1[) . Let u N be the optimal control, and X N the corresponding 
optimal state. Let u be the optimal control, and X the corresponding optimal state for the following 
finite horizon optimal control problem: minimize the cost, over all admissible controls, 

J(0,x,u) =E(P T X°' X ' U ,X°' X ' U ) + E [ T [(S S X°>*> U ,X°> X > U ) + \u s \ 2 ]ds. 

Jo 

Let us consider the so called stochastic Hamiltonian system 

d d d 

dX s = [A S X S ~ B s (B* s y s + ( D lY z l)] ds + EP^ S + D\{B* s y s + J2( D sT z s)]dW* , 



dy s = -[A* s y s + J2 {CI)* 4 + S s X s ]ds + 



k=l 

t<s<T, 



X t =x, 

k y T = P T X T , 

(3-8) 

where y, z % G K™, for every i — 1, ...,d. By the so called stochastic maximum principle, the optimal 
control of the finite horizon control problem is given by 



Let us consider the stochastic Hamiltonian systems relative to the optimal control u and to 
the optimal control u, and let us denote by (X N ,y N , z N ) and by (x,y, 'zj the solutions of the 
corresponding stochastic Hamiltonian systems. 

r T _ 2 d 

Lemma 3.6. E^* / [ ^/S~ S {X S - X?) + \B*(y s -y^) + Y, (D*)* {l? s - z^ l )\ 2 ]ds -> as N -» oo. 
Jt 



Proof. The proof is based on the application of ltd formula to (y t — y^ , X t — X 



ti Ft ®r-y$,X T -X¥)=E :F * 
= -E* J\\BI (y s - yf)| 2 + 



d{y s -y?,X s -X») 
d 2 

i=l 
2 



2^(S;(y s - y f),(^ 



))]ds 



i=l 



d „t 



d r T 



ds-E^^J 



S„ X s — X 



N 



ds. 



Since yr = PtXt and y^ = P^ X^ , we finally get 
E^ (P T X T - PtX^ : X t -X^ X 

d r T 



4=1 



(is. 



By adding and subtracting E^* (P^Xt,Xt — X. 



r N 

L T / J 



e^ < p^y ( x T - \ , x T - x T 



N 



( (P T - Pfi) X T , X t - X% 

d „T 



-E*jf \B;(y s -y^ + (Diy(zl-z^)\ 2 ds-E^J t \y/S.(x a -X 



N ' 



ds. 
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Since {Pfi (X T - X^j , X T - X^j > 0, and by definition ( \P T - Pfi) X T ,X T - X, 
for N sufficiently large, also 

2 



i=l 



ds + 



ds -> 



f 1 

'as N — > oo. In particular this means that E^* / 



.N\ 



as N — ► oo. 



□ 



As a consequence of the previous results we deduce the following: 

Corollary 3.7. Assume hypothesis \3.1\ and that (A, B,C, D) is stabilizable relatively to \/S. The 
process P is the minimal solution of the Riccati equation in the sense of definition[ 



Proof. Fix T < N and on [0, T] consider the Riccati equation 

d 

-dP t N = G (A t ,B t , C t ,D t ; S t ; P t N , Qf ) dt + Y^Q^dW} 



pN 
T 



P N (T). 



Then 



{P t N x, x) = (P?XF, X$) + E^ / [(S r X?, X?) + \u^\ 2 ]dr 
By lemma [3761 we deduce that 



(3.9) 



(3.10) 



Prp ( Xt — Xj^ ) . Xj 1 — X 



as N 



So E^' (PfiXfi, X%) -> E^* (PtXt, X T J as N -> +oo, since 

(P^Xf, ) - E^ (p^ (1 T - X?) , X T - X$) + 2{P$X T , X$) - (P^X Tl X T ) 



By the construction of P and by lemma [3~B1 we have that, letting N +oo in (|3.10p 

(P t x, x) = E^ (P T X T , X t ) + E^ J [(s r X r ,X r } + \u r \ 2 ]dr 

So (P, Q) is the minimal solution of the Riccati equation in the sense of definition 13.21 □ 

By the previous calculations, we can now solve the optimal control problem with infinite horizon, 
when / = 0. 

Theorem 3.8. If Al) — A2) hold true and if (A, B, C, D) is stabilizable relatively to S, given x S M™, 
then: 

(1) there exists a unique optimal control u G Lj, (f2 x [0, +oo) , M fc ) such that 

J QO (0,x,u) = inf Joo (0, X,u) 

tiGL^,(nx[0,+oo),R fe ) 

(2) The process P defined in lemma 3.4 is the minimal solution of the Riccati equation. 

(3) If X is the mild solution of the state equation corresponding to u (that is the optimal state), 
then X is the unique mild solution to the closed loop equation: 

-l 



dX t = 



E 



C; 



2=1 

Xq = X. 



Ax t -B t (i+j2(Dd* p t D tJ (ptBt+Y^(tf t Di+(ciyp t Diyj x 

X t -Dt^I + J2(Df)*P t Dl\ ^ t B t +J^(^ t Di+(CiYP t Dl)\ X 



dt+ 



(3.11) 
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(4) The following feedback law holds P-a.s. for almost every t: 

-l 



/ d \ / d \ 

I + J2(DiyPtDi) [p t B t +Y J (^ t D\+{C$)*P t D i M X t . (3.12) 

(5) The optimal cost is given by J oo (0,a;,u) = {Pqx,x). 

The proof of this theorem is similar, and more immediate, to the proof of theorem 15.21 which is 
given in detail in section 5. In particular we deduce that 

l Ft J {(SrX**'",^'") + \u r \ 2 ]dr. (3.13) 



(P t x,x) 



4. The infinite horizon dual equation 

We first introduce some definitions. We say that a solution P of eauation l3.4l is bounded, if there 
exists a constant M > such that for every t > 

\P t \<M P-a.s. 

Whenever the constant M t , x that appears in definition 13.31 can be chosen independently of t, then 
the minimal solution P is automatically bounded. 

Definition 4.1. Let P be a solution to 13.41 We say that P stabilizes (A, B, C, D) relatively to the 
identity / if for every t > and x € R" there exists a positive constant M , independent of t, such 
that 

/ + oo 
\X*' x \ 2 ,dr < M P-a.s., (4.1) 

where X t,x is, for s > t, the solution to: 

d 

dX%' x = H s Xl' x ds + ^KlXl^dWi, 



(4.2) 



X t ' = x 



From now on we assume that the process P is bounded and stabilizes (A, B, C, D) with respect 
to the identity /. 

Remark 4.2. It is possible to verify in some concrete situations that (A, B, C, D) is stabilizable 
relatively to the observations V~S and that P stabilizes (A, B, C, D) relatively to the identity /. 
Here we present the case when, for some a > 0, A and C satisfy 

(A t x,x) + ±(C t x,Ctx) < -a\x\ 2 , (4.3) 

for every t > and x € R™, then (A, B,C, D) is stabilizable relatively to the observations y/S 
uniformly in time. Indeed, by taking the control u = 0, applying the Ito formula to the state 
equation we get, for < t < s, 

|X*' X '°| 2 < \x\ 2 + 2E^ J (ArX^- , X*' x >°)dr + E F J (C r X*' x >°, C r X^: x - )dr 
< \x\ 2 - 2aE :Ft f \X r \ 2 dr. 



By the Gronwall lemma we get 
So for every < t < T 

/ + oo 
\y/SX^ x '°\ 2 dr < M x , 

where M x is a constant dependent on the initial condition x, but independent on the initial time 
t. So, according to definition 13.31 (A, B, C, D) is stabilizable relatively to the observations y/S, 
uniformly in time. Moreover, assuming that S > el, for some e > 0, by (|4.3|1 . we also get that P 
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stabilizes (A, B, C, D) relatively to the identity /. Indeed, by the previous calculations, denoting 
by X and u respectively the optimal state and the optimal control for the infinite horizon control 
problem with / = 0, it follows that 

/ + oo r+oo 
[\X r \ 2 + \u\ 2 ]dr <E* J \X^ xfi \ 2 dr < M x , 

which immediately implies (|4.ip . 

The Datko theorem holds also in this case. 

Theorem 4.3. If P stabilizes (A, B,C, D) relatively to the identity and it is uniformly bounded in 
time, then for the solution of equation \4-^\) there exists two constants a > 0, C > such that 



E ^ t \Xl' x \ 2 < Ce- a[s ~ t] \x\ 2 V-a.s. (4.4) 
Remark 4.4. Note that if X l s x is solution of equation 

d 

dx*' x ' r > = i l.x [■'■'■ + ^J/\:.v: '"r/u ; + <q a ds, 

X^ = x, 
then by the previous theorem 

» + oo / />+oo 



J E\Xl' x ^\ 2 ds < C (J E\ Vs \ 2 + \x\ 2 ) 



In order to study the optimal control problem with infinite horizon and with / 0, we need to 
study the BSDE on [0,oo), 



dr t = -H:r t dt-P t f t dt-Y,(Kl)* gldt + YsdtdWi, t>0 (4.5) 

i=l i=l 

where the final condition has disappeared but we ask that the solution can be extended to the whole 
positive real axis. We make the following assumption on /: 

Hypothesis 4.5. f is a process in L 2 V {Q x [0,+oo),M") n L^(n x [0,+oo),R"). 

Proposition 4.6. Let hvpotheses \3.1\ and \4-5\ hold true and assume that P is bounded and stabilizes 
(A, B,C, D) with respect to the identity I. Then equation |^.5[ j admits a solution (f,g) S L^ifl X 
[0,+oo),R") x L 2 v {tt x [0,T],R nxd ) 7 for every T > 0. 



Proof. For integer N > 0, we consider the BSDEs 

dr? = -H?r?dt - P t f t dt - J2 ( K l)*9t N dt + ^9i' N dWi, t e [0, T] 



(4.6) 



N — u - 



By proposition 12.81 we know that equation (|4.6p admits a unique solution (r N ,g N ) that belongs 
to L 2 , (n,C([0,iV],M n )) x L 2 , ([0,N] x 0,R" xd ), for every N E N. The aim is to write a duality 
relation, see remark [2T6| between r N and the process X N , solution of the following equation 

d 

dX» = H s Xfds + J2 KXsdWl , se[t,N] 



is 



By duality between r N and the process X , and by estimate (|4.4[) we get 



(Psfs-X^ds 

<cj^ ||P s |Uoc (n) ||/ s |Uoo (f2)e -t(-*)(E^|rf| 2 )^ S 

< - «-* ( — * ) l|3 5 -lli-<n)ll/-lli-(n)«fa + M-E^IrT | 2 , 
M Jt a 



2 1 

where we can take /i > such that /i— = — . So we get 

a 2 

|r,T < C, (4.7) 

where now C is a constant depending on a, P, /, but C does not depend on N. 

So also sup t>0 Elr^l 2 < C. By computing dlr^] 2 , see e.g. relation (|2.24|) and by the previous 

estimate we get for every fixed T > 0, 



E [ T J2\gl- N \ 2 d s <C, 
J ° i=i 



where C > does not depend on N. Then we can conclude that for every fixed T > there exists 
f and g such that r w r in L^fi x [0, T],R n ) 
every < t < T < N, by the weak convergence, 



f and g such that r w f in Z^fi x [0,T],R n ) and g N g in L^fi x [0, T], K nxd ). Moreover, for 



Jt Jt Jt i=1 Jt i=1 

By the Ito isometry, the stochastic integral J 4 T X^=i(ffs Ar )*(^W s l is bounded in L?r (fi), so it con- 
verges weakly in £^($1); since every T G (f2) can be written as T = Jq "fldW*, weak converges 
and Ito isometry imply that 

E [ iidw: f f2( 9 i' N ydwi=E J2^(gi' N Tds 

• • / J2^(9tTds = E jidWj J2(gl)*dWl, 



and this allows to say that 

/>T d i>T d 



Jt Jt „•_-, 



i=l i=l 

So the pair (f , g) is a solution to the elliptic dual equation (|4.5j) , indeed 

„X -X /.T ^ />T 



f T +/ H;? s d S + f PJ s ds+ f J2(KlYglds- f J2(gl)*dW:. 
Jt Jt Jt • i Jt ■ , 



i=l •"■ i=l 



Since T > is arbitrarily, (r,g) is defined on the whole [0, +oo). It remains to prove that r G 
L*,(n x [0,+oc),R n ). We set 

'f t < t < N, 
t > N. 
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So r)^ € L?p(£l x [0, +00), R n ). We write a duality relation, see (|2.18p between f and X*' solution 
of the following stochastic differential equation 

d 

dxf = H s xfds + KxfdWl + ryfds, 

xf=o. 

By duality we get 

E/ \f s \ 2 ds = E (PJ s ,xf)d s + E(f N ,X^ N ). 
Jo Jo 
Letting N — ► 00 on both sides we get on the left hand side 

Um A r . ro E / |f s | 2 cLs = lim E / |f s | 2 ds = E / \r s \ 2 ds, 
Jo Ar ^°° Jo Jo 

by monotone convergence. On the right hand side, by theorem 14.31 and estimate (|4.4j) . we get 
; r> r v;- N it ,. . \ ^ 



E/ (P,/,,.*? }ds + E<r JV) Xft 
Jo 

<l3aAr^ o-2-||i , lli-(nx[o,+oo))E / IM^+t^E J \xf \ 2 ds + ^E\f N \ 2 + — E|xf| 2 
<-2-||^lli-(nx[0,+oo))ll/llL»{nx[0, +O o))d* + Ci / \r s \ 2 ds + C + a,E j \f s \ 2 ds 



JO JO 

where we can choose C\ = <zi = 4, and C does not depend on AT. 
Putting together these inequalities we get 

\f s \ ds < C 2 ||-P|ll=o(n x [o !+00 ))||/|||2( 0x [ . +00 )) + 2C, 

and this concludes the proof. □ 



(i 



Remark 4.7. As a consequence of the previous proof, we get 

E|f T | 2 -> as 00. (4.8) 



Remark 4.8. Equation (|4.5j) has non Lipschitz coefficients and is a multidimensional BSDE thus we 
can not use the Girsanov Theorem, as done in [6], to get rid of the terms involving K. Moreover 
the typical monotonicity assumptions on the coefficients of this infinite horizon BSDE, see [7], are 
replaced by the finite cost condition and by the requirement that the minimal solution (P, Q) of 
(ll.lOp stabilize the coefficients relatively the identity, see definition 14.11 

5. Synthesis of the optimal control in the infinite horizon case 
We consider the following stochastic differential equation for t > 0: 

d 

dX s = (A S X S + B s u s )ds + {C\X S + D\u s ) dWl + f s ds s > t 

i=l 

X t = x, 

Our purpose is to minimize with respect to u the cost functional, 

r+cc 

J oo (0,x,u)=E l{S s X°> x ' u ,X°' x ' u ) + \u s \ 2 ]ds (5.2) 
Jo 

We also introduce the following random variables, for t E [0, +00): 



(5.1) 



J 00 (t,x,u)=E^ J [(S a X t a '*' v ,Xtr'' u ) + \u s \ 2 ]ds 
Throughout this section and the next section we assume that 
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Hypothesis 5.1. We will make the following assumptions: 

• S > el, for some e > 0. 

• (A, B, C, D) is stabilizable relatively to S . 

• The first component P of the minimal solution of the Riccati equation is bounded in time. 
Notice that these conditions implies that (P, Q) stabilize (A, B, C, D) relatively to the identity. 

Theorem 5.2. Let hypotheses ] 3. 11 \4-5\ and [5J\ hold true, then: 

(1) there exists a unique optimal control u £ L 2 ? (il x [0, +oo) , M fc ) such that 

J oc (0,x,u) = inf Joo (0, x,u) 

uGi^(nx[0,+oo),R' E ) 

(2) If X is the mild solution of the state equation corresponding to u (that is the optimal state), 
then X is the unique mild solution to the closed loop equation for: 



dX t 

d 

E 

i=l 
X = X 



A t X t - B t A(i, P t , Q t )X t + (J + £ (Di)* P t Di)-i(Btr t + £ (£>*)* g\) 



i=i 



dt + f t dt 



C\X t - D*(A(t, Pt,Q t )X t Ml + £ (DlYPtDl (B* t r t + 52 (Pi)* 9$) 



i=i 



dWl, t > 



(5.3) 



u t = 



(3) The following feedback law holds P-a.s. for almost every t > 0. 

/ d \ ~ X ( d \*d 

I + 52 (DlYPtDl P t B t + 52 (Q\D\ + (Cl)*Q t Di) X t +B* t r t +52{D\)*g\. (5.4) 

\ i=l /\ i=l / i=l 

(4) 77ie optimal cost is given by 

/>00 

J(0,x,u) = (P x,x) +2(r ,x) +2E / (r s ,/ s )rfs 



Jo 



Proof. Let us consider the sequence (X N , u N , P ) respectively the optimal state, the optimal 
control and the first component of the solution to the Riccati equation ()3.6p defined in section 3 
associated with the problem: 
minimize over all u 6 L^((0,N) x f2;R fe ) : 



J N {0,x,u)=E / [(S s X s ,X s ) + \u s \ 2 ]ds 
Jo 

where X is the solution to equation (|5.ip in [0, N]. 

And let us consider the minimal solution P evaluated at time N and the associated problem: 
minimize over all u 6 L^,((0,N) x r2;R fc ) : 

J(0 t x,u)=E / [{S s X s ,X s ) + \u s \ 2 ]ds + (P N X N ,X N ). 



where X is the solution to equation (|5.f p . It turns out from Theorem 12.91 that the optimal state 
is the solution to equation (|5.3p considered in [0, N] and that the optimal control is given by (|5.4p . 
Hence arguing exactly as in Lemma 13.61 we have that for every T < N , 

a) u N -> was TV -> +oo in the space Lf,((0, T) x Q; M fc ) ; 

b) X N — > A" as N — ► +oo in the space Lp((0, T) x O; R n ) and then using a) and the Gronwall 
Lemma, X N -> X as N -> +oo in the space Zp(f2; C([0, T]; K™)); 
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By computing d(P^X^, X, ) + 2(r^ , X^) , where r N was defined as the first component of the 
solution (r N ,g N ) of equation (|4.6|) we get, 

N vN\ i |-jV|2i > _ / D JV„ „\ i rwN ™\ OTP / /„JV 



E [(S s X?,X?) + \u?\ 2 }d S = {P N x,x}+2{r»,x)-2E (r?,f s )ds 
Jo Jo 

- E / o iu+E(^)* pAr ^J (#>f +E(^)*^ >JV )i 2 ^ (5.5) 

and so being P w and r N uniformly bounded, see also (|4.7|) and / G Lp (fl x [0, +00), R"), we get: 

E / [(S.X?,X?) + | 2 ]ds < E / \(S.X?,X?) + |zzf |> < C 
Jo Jo 



and 

E 



y iu+E(^)* pAr ^J +E(^)*^)i 2 ^<c 



We get, passing to the limit: 



and letting T — > +00: 



E / [<S S X S ,X S ) + |u s | 2 ]ds < C 
Jo 



+00 



E [(5 s X s ,X) + |u,| 2 ]ds<C (5.6) 
Jo 

Therefore using estimate (|5.6p and the equation (|5.ip we get: 

supE|X T | 2 <C (5.7) 

T>0 

Now we consider the fundamental relation in [0, T\. 

E^ [(S.X?,X? ) + |«f | 2 ]ds = (Pq^,^) - (P^X T ,X T ) + 2^) - 2(4, X T ) - 2E* j\r»,f s )ds 

~ E £\ (i + ^(Diy P N sD^ (P;rf + E(^)* 5 ^)| 2 d S , (5.8) 

and we notice as first that since all the other terms converge as N ^ +00 we do have, recalling also 
that r N f in L|,(fi x (0,T);K n ) and 3^ 5 in Z£(fi x (0,T);M fc ), 

lim E f T \(l + J2m*P N sDl] (B s *rf + E(^)*3^)| 2 d S = 

E / T 1 ( 7 + E (^)* W) + E (^)* ^)i 2ds 

Hence letting tend to +00 in (|5.8|) we get: 

E^ [(5 S X S ,X S ) + |u s | 2 ]ds = (P x,x) - (P T X T ,X T ) + 2(f ,x) - 2(f T ,X T ) - 2E^ J (f„f a )ds 

~ E £\ ( / + E(^)*^) (P;f s + E(^)*ff:)| 2 ^ (5-9) 
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Therefore, since (PtXt, Xt) > 0, and also by (|5.7|) and (|4.8[) we obtain that 

p + ca r+oo 

E [{S s X s ,X s ) + \u s \ 2 }ds<{P x,x) + 2(f ,x)-2E (f s J s )ds 
Jo Jo 

p+co / d \ 1 d 

••I i(i+E(^:)*w) (i?;f s +^(^)*g:)i 2 d S , (5.10) 

Now we need to prove the opposite inequality. We can reduce to an admissible control such that: 

" + 00 



-00 



r-l-oo 

>J(0,x,u)=E / [\^S S X^ X <°\ 2 + \u s \ 2 ]ds 
Jo 



Hence we have that: 

hoo r+oo 



-00 



p + oo r-t-oo 

>J(0,x,u)=E [\Vs s X^ x -°\ 2 + \u s \ 2 ]ds>E [e\X^ x -°\ 2 + \u s \ 2 ]ds 
Jo Jo 

r u.x.O 



Thus again the same estimate holds for the state variable X u ' x ' := X, 



supE|X T | 2 <C (5.11) 

T>0 



Computing d(P"X s ,X s ) + 2(f s ,X s ) we get that: 



E [(S s X s ,X s ) + \u s \ 2 }ds = E(P Q N x,x)+2(f a ,x)-2E (f s J s )ds 
Jo Jo 

r N ( d \ 1/2 / d 

+ E / o \{i+J2( D sT p s ND ij u + a+E^r^r 1 * 

J2(Q^ N Dl + (CiyP s N D^\ X s + B:f s +J2Dl(glA \ 2 ds 

i=i / t=i / 



* P?B. 



fN ( d \ 1 d 

] J Q \\l + J2(Dl)*P s N Dlj (B:f s +J2(Dl)*9l)\ 2 ds. (5.12) 



Now we observe that 
E 



r N ( d \ d 

/ | (l + J2(Dt)*P s N Dt) (B^+^iD^dstfds 
J o \ i=l / i=l 

/■+oo ( d \~ d 

= e I i U+E^r^J (s>- s +E(^)*^)i 2ds 

/.+00 / d \ ~ d 

/ i u+Ewrw (^.+EW)V,)I , a 1 

V »=i / »=i 



and by the Dominated Convergence Theorem we have, 



r +oo ( d \ d 

W I I + E^W W.+£MV.)| 9 * 

V i=l / i=l 

/-+oo ( d \ ~~ d 

^ E J I U+E^)*^) (s;f.+E(^)V.)i a ds. 



2ll 



Moreover 



r + oo / d \ 1 d /-+oo d 

/ | (l + J2(DiyP s N Di) (B* s r s + J2(Di)*9i)\ 2 ds< / |(B;f fl + £ (D*)* pj)| 2 rf S , 

V i=i / i=i Jjv i=i 



and, from (|5.9p we have: 



/ |( J B;f s + ^( J D:)*g:)| 2 d S < 
Ja i=i 



So we can conclude: 



lim E 

AT— >+oo 



/.+00 / d \ 1 ci 

/ i / + £ (^)* pffl: + £ (x>*)*£)i 2 d* = o 



So, letting TV — ► +oo in flETH) we get for every admissible control u 

r+oo 

E / [(S S X S ,X S ) + \u s \ 2 ]ds > E(P x,x) +2E(f ,x) - 2E / (f.,/,)da 



(.+oo ( d \ 1 d 

-E/ I /l^(J)if^ (i^f s £)| 2 d S . (5.13) 
■'Q V i=l / i=l 

Then from (|5 . 1 0[) and (|5 . 13[) the theorem easily follows. 



□ 



6. Ergodic control 



In this section we consider a cost functional depending only on the asymptotic behaviour of the 
state (ergodic control). To do it we first consider a discounted cost functional that fit the assumptions 
of section 5 and then we compute a suitable limit of the discounted cost. Namely, we consider the 
discounted cost functional 



J a (0,x,u)=E I e- 2as [(S s X Q s ' x ' u ,X Q s ^ u ) + \u s \ 2 }ds, (6.1) 
Jo 

where X is solution to equation (|5.ip . with A, B, C and D satisfying hypothesis 13.11 and / G 
L^p (fi x [0, +oo), W 1 ). Moreover we recall that we assume hypothesis 15. II When the coefficients are 
deterministic the problem has been extensively studied, see e.g. [2] and [21] . 

Our purpose is to minimize the discounted cost functional with respect to every admissible control 
u. We define the set of admissible controls as 

U a = juei£(fi x [0,+oo),R fc ) : E e- 2as [(S s X°' x ' u , X°' x ' u ) + \u s \ 2 }ds < +oo| . 

Fixed a > 0, we define Xf = e~ as X s and u" = e~ as u s : we note that if u G U a , then u a E U. 
Moreover we set A<$ = A s - al and /« = e~ as f s , and f a G Z^(f7 x [0, +oo)) n L^(fl x [0, +oo)). 
X" is solution to equation 

d 

dX« = {A«X« + B s u a s )ds + Y.( Cl s X s+ D > a s)dWl + ffds s>0 (g2) 

4 — 1 

X$ = x, 

By the definition of X a , we note that if (A, B,C, D) is stabilizable with respect to the identity, 
then (A a , B,C, D) also is. We also denote by (P a ,Q a ) the solution of the infinite horizon Riccati 
equation (|3.4p . with A a in the place of A. Since, for < a < 1, A a is uniformly bounded in a, also 
P a is uniformly bounded in a. Now we apply theorem l5.2l to the control problem for the discounted 
cost J a . Let us denote by (r a ,g a ) the solution of the BSDE obtained by equation (|4.5p . where / is 
replaced with and H and K are replaced respectively by H a and X". H a and isT a are defined 
as in (|2.9p . with ^4 Q and P Q respectively in the place of A and P. 
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Theorem 6.1. Let huvotheses VS .1\ and \4-5\ hold true; assume that f £ L^(£l x [0, +00), R n ) then: 

(1) there exists a unique optimal control u a £ lA a such that 

J a (0,x,u Q ) = inf J a {0,x,u) 

(2) The following feedback law holds P-a.s. for almost every t > 0: 

< = - (r+J2( D ?)* p t a vt) ip f a B t +j2(QrDi + (ciy p?Di)\x+B;r?+Y,Di( 9 ry, 

\ i=l ) \ i=l / i=l 

(6.3) 

where X is the optimal state. 

(3) The optimal cost J a (0,x,u a ) := J (x) is given by 



J (x) = (P£x, x) + 2(r£, x) + 2E / (rf, ff)da 

Jo 

-E / \(I + J2( D ?)* P?D\y\B* t r? +J2( D tY 9n\ 2 ds. (6.4) 

Jo i=i i=i 

The optimal cost J a {x) — > +00 as a — > 0. We want to compute lim Q ^o aJ Q (a;). In order to 
do this, we need some convergence results, the first concerning the Riccati equation. To prove 
this convergence, we note that, by applying the Datko theorem, we are able to prove estimates 
independent on a. 

Remark 6.2. By the Dakto Theorem, see theorem 14. 3[ we can prove an exponential bound for the 
process X a,t - X which solves the following equation 

d 

dx ^t,x = (H«)X^ x ds + (Kf^)* Xf^ x dW l s , s > t 

i=l 

X t =x 

We can conclude that there exist C, a > 0, independent on a such that for every s > t: 

E^|X A "| 2 < Ce- a(s ~ l) \x\ 2 P-a.s. (6.5) 

Lemma 6.3. Assume that hypothesis VJ . 1\ holds true, that f S L^(Vl x [0, +00)). Then P" — > Pt as 
a — > for all t > 0, where P is the minimal solution of the BRSE. 

Proof. We can consider the case t — without loss of generality. 

Since (Pqx,x), respectively (P$x, x), is the optimal cost of the linear quadratic control problem 
with state equation given by (|5.1[) . respectively by (|6.2p . in the particular case of / = 0, and cost 
functional given by (|5.2p . respectively by (|6.ip . we immediately get that 

P " < P for all a > 0. 

Moreover we get that 

/■+00 

(P°x,x) = E / [(S s X a (s),X a (s)) + \u a (s)\% 
Jo 

where 

2° = - + E ( D i)* P t aD ^J ( p ? B t + E (^ D l + (Cl)*P t a Di)^ X?, 

and X a is the state corresponding to the control u a . So the pair (X a ,u a ) is bounded in L^,(fl x 
[0, +00)) x Lj,{Q, x [0, +00)), so there exists a sequence ctj — > as j — > +00 and a pair (X,u) 
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such that (X a i ,u a i ) — * (X,u) in L^(£l x [0, +00)) x x [0, +00)). As a consequence of this 

convergence, the process X is solution to equation (|5.ip , with control u. So we get 



(Pox, x) <E [(S s X(s), X(s)) + \u(s)\ 2 } 
Jo 

r+oo 

< lim^ +00 E / [(S s X^(s),X^(s)) + \u a > (s)\ 2 } = ljm^ +QO E(P^ x,x). 

JQ 

□ 

We remark that we can exploit a sort of separation principle, typical of the linear quadratic case, 
that allow to estimate separately the quadratic part from the linear part. Next we want to prove 
that, as a — > 0, the optimal pair for the discounted control problem, that we denote by (X a ,u a ) as 
in the previous proof, converges to the optimal pair (X,u), defined in theorem 15.21 

Lemma 6.4. Assume that hvvothesis \3. 1\ holds true, that f 6 (£1 x [0, +00), R k ). Then, for every 
T > 0, X a —>X~ in Lp(fi x [0,T],M") and u a -*u in L^(fl x [0,T],R fe ) asa->0. 

Proof. We consider the stochastic Hamiltonian system (|3.8|) and the stochastic Hamiltonian system 
for the discounted problem 

d d d 

dXf = [A%X?-B s (B* s y a (s)+J2 W z^)]ds + ^RI S » + D\{B* s y« -^(D*)* z? k )]dW* + ffda, 

»=1 i=l k—l 

d d 

dyf = -[(Af)*yf +]T ( C 'sY z s" + SsX?]ds + ^z^dW*, t<s<T, 

i=l i=l 

X? = x, 

..a _ pa va 
K i)T ~ r T T' 

(6.6) 

Proceeding as in lemma EPH we get 

^ (Vt ~ VT, Xt ~ Xt) = f a(yf,X s ) - a(y s ,Xf)ds + j\f« - f„ yf - y s )ds 

-E F r^iXa-X^ds-E^ [ T \B* s (yf-y s )+Y,m*(z^-zl)\ 2 d S , 
Jt Jt i=1 



that is 



F * (P?(X« - X T ), X« - X T ) + E r ({P^ - P T )X Tl X« - X T )=E F J a(y s Q - y s , X«)ds 
- j\(yf,X?-X s )ds - E^ j T \B* s (y« - y.) + - z\)\ 2 ds. 



S,(X? - X s )\ 2 ds + E * / (/ S Q - f s ,y« - y s )ds 
t Jt 

It follows that 

E^* [ T \^S~ s (X«-X s )\ 2 ds + E^ [ T \B* s (y°~y s ) + Y,(Din^ 
Jt Jt ■ 1 



i J\|2, 



i=l 

as a — > 0. □ 
Finally we need to investigate the convergence of r a to r, where (r, g) is the solution of equation 

623). 

Lemma 6.5. For all fixed T > 0, r a \[ ,t]-> r | [0 ,t] in L 2 (tt x [0,T]). 
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Proof. First we note that f a is uniformly bounded in a and 

,T ,T 

E J \H?\ 2 dt + E^ J \K?\ 2 dt < C, (6.7) 



where C is a constant depending on T, x, A, B, C and D, but not on a. So, see proposition! 
equation (|4.5p , where / is replaced by and _ff and K are replaced respectively by H a and if™ 
admits a solution (r Q , ff Q ) G Z£(n x [0, +00), K") x L^(Q, x [0,T],M" xd ), for every T > 0. Now let 
us consider ?y £ L|,(f2 x [0, T], M. n ). rj can be defined on the whole halfiline [0, +00): we set r\ t = for 
t > T. Let X t,x ' n be the solution of equation (|2.17p and let X a ' t,x,n be the solution of an equation 
obtained by equation (|2.17p by replacing H with H a and K with K a . By relation (|2.18[) . we get 

P T r + °° rp ya, 0,0.,, „ 

(r^,r] s )ds = E (Pj7 s a , X^°^ r >)ds + E / (P s Q /f,Xf' ' T '°)ds. (6.8) 



and also 

i>+OG 



:jf (f s , Vs )ds = E^ (P s f s ,X^)ds + E^ (p s f s ,xJ' X °'°"'' }ds. (6.9) 



By theorem l4.3l and by lemmas 16731 and l6T4l thc right hand side in Ij6.8|> converges to the right hand side 
of ([5T9"]). So we get that r a \[ 0>T ]^ r \ [0 . T] in L^(Q x [0, T], R"). In order to get that r a \[ ,t]^> r | [0 ,t] 
in L 2 V (VL x [0,T],K"), it suffices to prove that \\r a | [0)T] || i 2 j (n X [o,T],K») Ik I[o,t] IIl^(Qx[o,t],b»)- 
We take in (J5T5J) r\ t = rf for < t < T. We get 

rT nT r-\-oo a r a 

E \r«\ 2 ds = E (P°f?,X?> '°> ra )ds + E (P"f°, X"' T ' Xt ' '°)ds. 



By remark 14.31 and by lemmas 16.31 and 16.41 the right hand side converges to 

pT /-+oo r 

E / (P s f s ,X s < ^°)ds+E / T\J\.X! X - ' ' '°)ds, 
Jo Jo 

and this concludes the proof. □ 



Remark 6.6. Following the proof of proposition 14.61 it is easy to check that there exists a constant 
C > 0, independent on a such that for every t > 0, \rf\ < C. 

— ct 

We can now study the convergence of aJ . 

Theorem 6.7. Assume that hypothesis lff.il holds true, that f £ -L??(f2 x [0, +00), W 1 ). Then 

r+00 

hm^oaAz) = lim Q ^ 2«E / (r?,f?)ds 

Jo 

p+oc d d 

Hrn^oaE / |(I + £ (£>*)* P?tf a )-\Btf + £ (£>*)* gf^ds 



Proof. For every a > 0, by theorem 15.21 we get 
J Q ( 2 ;) = (PoX, 2 ;)+2(r«,x) + 



/>00 /"OO 

J o Jo 1=1 i=l 



So, since P a and r™ are uniformly bounded in alpha, for every a > 

poo d d 

aE / KJ + ^^^Dy-^S^+^^^ni^^C, 
- 70 <=i f=i 

where C is a constant independent on a. We can conclude that, 

r+00 

Vm a ^ aJ a (x) = hm Q _ 2aE / (rf, ff)ds 

Jo 

p+oc d d 

- m a ^oaE / I (J + £ (£>:)* P^Di)-\B* s ^ + £ S?' 4 )!^ 

- 70 i=l i=l 
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